Situated Speech Understanding

Speech and language do not occur in a vacuum much of the meaning of an utterance comes from its context, including when and where it was uttered, what the person saying it was doing at the time, who was there to listen to it, and why the speaker decided to speak in the first place. In my research, I aim to unify a number of disciplines such as speech recognition, discourse modelling, plan recognition and natural language understanding into a single coherent framework that deeply understands situated speech. To do so, such a framework has to interact with the world it shares with other language users and model and understand this world in ways similar to theirs to establish a shared context. In this proposal, I describe the elements of such a framework that shares a virtual game playing world with several human players. The framework models the progress of the game in terms of the physical situation, the discourse situation and the players’ plans and goals and grounds speech uttered by the players in terms of these rich situational